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ABSTRACT 

Bladder cancer is widely spread in the world. Many adequate diagnosis techniques exist. Video-endoscopy 
remains the standard clinical procedure for visual exploration of the bladder internal surface. However, video¬ 
endoscopy presents the limit that the imaged area for each image is about nearly 1 cm^. And, lesions are, 
typically, spread over several images. The aim of this contribution is to assess the performance of two mosaicing 
algorithms leading to the construction of panoramic maps (one unique image) of bladder walls. The quantitative 
comparison study is performed on a set of real endoscopic exam data and on simulated data relative to bladder 
phantom. 

Robustness and accuracy 
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1. INTRODUCTION 


The applicative aim of this contribution concerns bladder cancer detection in image sequences recorded during 
endoscopic examinations. The 2-D cartography of an image sequence, also called image mosaicing, relies on 
a prior registration of consecutive image pairs of the video sequence, and then on the superposition of all the 
images onto a single common panoramic image. Lesion detection and evolution assessment may be far easier in 
such mosaics than in isolated images showing, each, only a very small part of the region of interest. Mosaicing of 
human organ images is a few treated problem (se^i® for applications of mosaicing in mammography, angiography, 
ophthalmology, and microscopy), the existing solutions being not automated or needing a priori knowledge like 
sensor position and being only able to register few images. In the case of bladder endoscopy, image mosaicing is 
difficult for several reasons. First, image primitives are not easy to extract robustly (e.g., contours), and their 
background is severely textured. Moreover, the recorded images have a great inter- and intra-patient variability. 
Second, the endoscope position is unknown during the image acquisition, since urologists can move “freely” 
the instrument inside the bladder. Third, a video sequence consists generally of thousands of images. One of 
the technical question for the consecutive registration of pairs of images is : how to register robustly, precisely 
and with an acceptable computation time all the images of a sequence? The computation time may be the 
less critical factor since the mosaic must be available for a further diagnosis which is usually performed some 
dozens of minutes or hours after the examination itself. In this paper, we focus on the registration of consecutive 
images, denoted by Ik (target image) and Ik+i (source image), where k stands for the image index in the video 
sequence. The registration of Ik and Ik+i consists in finding a 2-D/2-D perspective transformation T{x,y\0k) 
which superimposes Ik+i on Ik- In notation T{x,y;0k), {x,y) represents a 2-D point in the domain of image 
Jfc+i and Ok is the set of a^- parameters of the perspective transformation (related in eq. ([T]) to the translations 
tx and ty, plane rotation (f), scale factor /, shearing parameters Sx and S'y, and perspective parameters 031 and 
032 ). The perspective transformation {x',y') = T{x,y,9) of the 2-D space reads : 
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and involves 8 independent parameters (033 = 1). The registration of images Ik and Ik+i is stated as the 
maximization of a similarity criterion of the form: 

Ok = argmax 5(7fc, r(7fe+i;0)). ( 2 ) 

eeR» 

The difference between different registration algorithms to be chosen to solve this problem lies in the choice of 
the measure of similarity S and in the choice of the numerical algorithm of optimization. 

2. IMAGE REGISTRATION ALGORITHMS 

The bladder images do not systematically include image primitives (e.g., corners or contours) that can be robustly 
enough extracted.^ For this reason, the most simple registration methods relying on the segmentation of an image 
primitive cannot be used, and we must consider a great number of image pixels when choosing the measure of 
similarity S{Ik, T{Ik+i-,0)). 

2.1 Aqd : Quadratic distance based algorithm 

The first algorithm is based on a measure of dissimilarity Sq^iilk, T{Ik+i', d)) defined as the quadratic 

distance between the grey levels of the pixels of Ik and these of the perspective transformation of the pixels of 

7fc+i : 


SQu{Ik,T{Ik+i-,9))= Y. [Ikix,y)-Ik+i{T{x,y;e)f (3) 

{x,y)GlknIk+i 


where (x, y) denotes the coordinates of a pixel common to both Ik and T{Ik+i', 9) images. The minimization 
of this measure can be done using Baker and Matthews’ inverse composition algorithirP whose goal is to estimate 
the optical flow, i.e., the apparent motion between two given images. 

2.2 Ami : Mutual Information based algorithm 

The second algorithm is based on Viola and Wells’ approach EMMA^SI (EMpirical entropy Manipulation 

and Analysis). Ami aligns images Ik and 7^+1 by maximizing the measure of similarity Smi^Iu, T{Ik+i', 9)) 
defined as the mutual information between Ik and T{Ik+i]9). Shortly speaking, the mutual information is a 
statistical measure computed with the grey level entropies H{Ik) and H(T{Ik+i;9)) of the overlapping parts of 
Ik and T{Ik+i]9) and with the joint entropy H{Ik,T{Ik+i\9)) : 

SMi{Ik. T{Ik+i\9)) = H{Ik) + H{T{Ik+i,9)) - H{Ik,T{Ik+i,9)) (4) 

This measure is used together with a stochastic descent gradient algorithm in the optimization process of 
eq. dll). The mutual information is well suited to the registration of textured images.^ 

3. COMPARATIVE STUDY : EXPERIMENTS AND RESULTS 

In this section, we present the registration results obtained with both measures of similarity applied on com¬ 
mon data sets obtained from real human bladder examinations and simulated endoscope displacement and on 
simulated data from a realistic phantom constructed using a pig bladder cartography. 
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Figure 1. I, II and III : Three reference images extracted from a real endoscopic exams for robustness evaluation tests. 
The chosen images present both texture and illumination variabilities. 

3.1 Robustness evaluation 

Three images with very different visual aspects (various textures and illumination conditions) were extracted 
from human endoscopic sequences to assess the robustness of the algorithms (see Figure [IJ. These three images 
were all taken as reference images {Ik target images in eq. @). Ik+i source images were computed by applying 
known simulated 2-D transformations on the Ik target image {Ik+i = T{Ik+i]d)), as if we simulate a real 3-D 
displacement of the endoscope. The 3-D displacement includes two translations corresponding to and ty in 
eq. (P while tz relates to the / scale factor and 3-D rotations (in plane rotation </) in eq. © and two out of plane 
rotations ^ and a related to 031 and 032 . In this way, it is possible to compare the calculated transformations 
with the known transformations already used to simulate images. 

These {Ik, Ik+i) image pairs allow for an assessment of the largest endoscope viewpoint change leading to 
successful registrations. The parameter value intervals for which a successful registration was obtained are de¬ 
tailed in TabU] For the Ami algorithm, intervals are : = ty = ±30 pixels, / = ±25%, (j) = ±20° and a =■0= 

±20°. These limits are more restricted for the Aqo algorithm : tx = ty = ±25 pixels, / = ±15%, (p = ±10° 
and a = V’= ±10°. Even if for both methods, the translation limits are roughly the same order (with a slight 
advantage for the mutual information algorithm Ami), the robustness is clearly better for the mutual information 
method in terms of scale factor changes and in- and out of plane rotations. 



Transformation value intervals 

Transformation parameters 

Aqd 

Ami 

Real endoscopic exam 

Translation {tx and ty) 

±25 pixels 

±30 pixels 

±5 pixels 

Scale factor (/) 

±15% 

±25% 

±2% 

In plane rotation {(p) 

±10° 

±20° 

±1° 

Out of plane rotations {tp and a) 

±10° 

±20° 

±1° 


Table 1. Transformation value intervals for with a successful registration was obtained for both Aqd and Ami- The last 
column designate transformation value intervals in real endoscopic exams in most cases (90%). 


3.2 Accuracy evaluation 

A quite realistic phantom was built using an excised pig bladder in order to test the registration accuracy of both 
methods. The pig bladder was incised, opened out and photographed with a camera. The pig bladder texture 
(see FigurePa)) is very similar to that human bladder. The area covered by the acquired picture is a 16 cm side 
square. The first image was taken in the upper left photograph corner. The other images of the sequence were 
obtained by simulating successively 10 pixel horizontal translations (14 upper images), a combination of 10 pixel 
translations and of 2° in plane rotations (upper 10 vertical images on the right photograph side), combination 







of 10 pixel translations and of 5% scale factor changes (lower 10 vertical images on the right photograph side), a 
combination of 10 horizontal translations and of 4° out of plane rotations (first 10 lower images from the right 
photograph side), etc. 



Figure 2. (a) Pig bladder photograph : the boxes indicate the simulated image sequence, i.e., the acquisition path, (b) 
Mosaic (map) image obtained with the mutual information algorithm using registration of successive images. The map 
is visually coherent, all textures being continuous from one image to another in the map. This map visually matches the 
image of the pig bladder photograph, (c) Same results for the optical flow method. 


All image pairs (Jfe, Ik+i) were registered with both methods. The ek,k+i registration accuracy criterion 
is defined as the mean distance between homologous pixels of the target images Ik and the registered images 
T{Ik+i',0k). This criterion is ideally equal to 0. 

In the case of simple translations (efc,fc+i ~ 0.2 pixels) and a combination of out of plane rotations (perspective 
changes) and translations (efc,fc+i ~ 0.6 pixels), the registration errors are equal for both methods. These errors 
are very small and imperceptible (see Figures [2Kb) and|2](c)). For the combinations of translations and in plane 
rotations, the errors are again equal for both methods (efc,fc+i ~ 3.5 pixels) (see Figure jS]). As we observed 
visually (Figure^]), these errors rather correspond to a small T{Ik+i\ Ok) image distortion without affecting the 
global mosaic (map) coherence. Especially in the map regions including image borders, the textures are without 
discontinuities. As shown in Figure |21 registration mean errors Ck.k+i values are equivalent for both algorithms 
in most sequence parts except in the part where the scale factor changes (images number 20 to 30) and for which 
Aqd algorithm is more efficient {jk,k+i ~ 1-5 pixels compared to 4.5 pixels). Again, these errors do not affect 
the global visual map coherence. It is noticeable that, due to the image acquisition rate (25 images/second) and 
to the small endoscope displacements (few millimetres/second), the real rotation parameters (< 1°), translation 
parameters and ty <5 pixels) and scale factor (< 2%) changes are in fact by far smaller than those imposed in 
our experiments. In practice, both methods led systematically to sub-pixel errors for limited and more realistic 
displacements. 

Figures 0] and 0 show two panoramic images constructed from real cystoscopic examination images using 
Aq£) and Ami respectively. The panoramic image in Figure 0] is a 1479 x 1049 pixel image constructed from 
a 450 image sequence using Aqd. In this panoramic image, two polyps are visible on the top-right and at the 
bottom left of the image. Both polyps can be accurately located in relation to each other. Figure [^represents a 
650 X 182 pixel panoramic image constructed from a 500 cystoscopic image sequence using Ami- There are no 
visible discontinuities on texture affirming a quite good visual coherence. 

3.3 Mosaicing speed 

Ami and Aqd were programmed in C language using OpenCV vision library. The evaluation of both algorithms 
robustness and accuracy was done using an Intel Dual core(TM) 2.40GHz, 2Gb RAM computer. The optimization 
method of the Ami algorithm requires, in average, 250 iterations to register consecutive images. And each image 
pair registration takes between 50 and 60 seconds. In Figure [3 the construction of the panoramic image took 
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Figure 3. Registration mean errors ek,k+i for both Aqd and Ami- efe,fe+i values are equivalent for both algorithms in most 
sequence parts except in the part where the scale factor changes (images number 20 to 30) and for which Aqd algorithm 
is more efficient. 


nearly 8 hours 27 minutes. However, in the same experimental conditions, Aqd is about 100 times faster than 
Ami- In fact, a mean number of 12 iterations was needed by the optimization algorithm of the Aqd algorithm 
to register a pair of images. The time of registration for an image pair varied between 0.3 and 0.6 second. The 
panoramic image shown in Figured was constructed in 3.20 minutes. The computation time of the Aqd makes 
possible the construction of partial panoramic image of the bladder during the standard cystoscopic examination 
procedure. 


4. CONCLUSION 

In terms of accuracy, both registration methods give comparable results with a slight advantage for Aqd- 
However, the Ami method is more robust than the Aqd method, while the computation time of Aqd algorithm 
(some tenth of seconds to register two images) is about 100 times smaller than that of the Ami algorithm. Future 
work will aim at combining both methods to reach the robustness of the mutual information method and to tend 
towards the computation times of the Aqd algorithm. 
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Figure 4. A 1479 x 1049 pixels panoramic image constructed from a 450 image sequence using Aqd- Two polyps are 
visible on the top-right and at the bottom left of the image. In this panoramic image, both polyps can be accurately 
located in relation to each other. 



Figure 5. A 650 x 182 pixels panoramic image constructed from a 500 image sequence using Ami- In this panoramic 
image, there are no visible discontinuities on texture affirming a quite good visual coherence. 
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